Appearance inspection apparatus and appearance inspection method

ABSTRACT

When a machine learning network trained with both a non-defective product image and a defective product image is used, it is possible to stably exhibit high detection capability for the defective product image having an unknown defect while shortening a takt time during the operation time. A processor executes a first learning process of causing a machine learning network to learn a non-defective product image added with a noise, and a second learning process of causing the machine learning network to learn a defective product image, and detects both an unknown defect having a characteristic different from a characteristic of the non-defective product image and a known defect having a characteristic designated as a defective site, by inputting a workpiece image to the machine learning network whose parameter has been adjusted by the first learning process and the second learning process.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims foreign priority based on Japanese PatentApplication No. 2021-190172, filed Nov. 24, 2021, the contents of whichare incorporated herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The disclosure relates to an appearance inspection apparatus and anappearance inspection method for inspecting an appearance of aworkpiece.

2. Description of Related Art

For example, Japanese Patent Application Laid-Open No. 2019-204321discloses a processing apparatus that determines whether a workpiece isa non-defective product or a defective product using machine learning bya computer.

The processing apparatus of Japanese Patent Application Laid-Open No.2019-204321 is configured to be capable of making a determination onwhether a workpiece to be determined is a non-defective product or adefective product using a non-defective product learning model and adefective product learning model by conducting supervised machinelearning for non-defective product data to generate the non-defectiveproduct learning model, and conducting supervised machine learning fordefective product data to generate the defective product learning model,and then, inputting data of the workpiece, and such an apparatus is alsocalled a workpiece appearance inspection apparatus.

Meanwhile, a defective product is hardly generated at a workpieceproduction site, and thus, there is a circumstance that it is easy tocollect a large amount of non-defective product data, but it isdifficult to collect defective product data. Therefore, non-defectiveproduct learning targeted for the non-defective product data is assumedas a solution, but in a case of a machine learning network trained onlywith the non-defective product data, a detection capability of adefective product is insufficient, and an inspection with a high degreeof difficulty is inferior in performance as compared with defectiveproduct learning.

Even if a large number of pieces of defective product data can becollected, a machine learning network that has trained with defectiveproducts exhibits a high detection capability with respect to defectdata taught at the time of learning, but detection with respect tounknown defect data is unstable, so that detection omission tends tooccur.

Therefore, a method of generating the non-defective product learningmodel trained with non-defective product data and the defective productlearning model trained with defective product data, performing aninference process on a workpiece image using each of the non-defectiveproduct learning model and the defective product learning model duringthe operation time, and combining obtained inference result isconceivable as disclosed in Japanese Patent Application Laid-Open No.2019-204321.

However, in a case where the non-defective product learning model andthe defective product learning model are used, tuning at the time oflearning is required for each of the two models, and tuning is alsorequired for the process of combining the two inference results, andthus, a learning difficulty level increases and the labor at the time oflearning also increases.

In addition, when an appearance of a workpiece is inspected using thenon-defective product learning model and the defective product learningmodel, a processing time becomes long since the inference process isperformed in each of the two models, and moreover, the process ofcombining the two inference results also requires time, so that anincrease in takt time may become a problem.

Furthermore, non-defective product learning and defective productlearning originally have different properties, and thus, there may be acase where it is difficult to construct logic that absorbs thedifference in properties between the non-defective product learning andthe defective product learning.

SUMMARY OF THE INVENTION

The disclosure has been made in view of the above points, and an objectthereof is to enable stably exhibition of high detection capability fora defective product image having an unknown defect while shortening atakt time during the operation time by using a machine learning networktrained with both a non-defective product image and a defective productimage.

In order to achieve the above object, in one embodiment of thedisclosure, it is possible to assume an appearance inspection apparatusincluding: a storage section that stores a machine learning network; anda processor that inputs a workpiece image obtained by capturing aworkpiece, which is an object to be inspected, to the machine learningnetwork and determines quality of the workpiece based on the inputworkpiece image. The processor is configured to be capable of executinga first learning process of adding a noise to a non-defective productimage corresponding to a non-defective product, causing the machinelearning network to learn the non-defective product image added with thenoise, and adjusting a parameter of the machine learning network suchthat a portion corresponding to the noise is extracted. In addition, theprocessor is configured to be capable of executing a second learningprocess of causing the machine learning network to learn a defectiveproduct image corresponding to a defective product having a defectivesite and adjusting the parameter of the machine learning network suchthat the defective site designated in advance by the user is extractedon the defective product image. Further, the processor is configured tobe capable of executing a process of detecting both an unknown defecthaving a characteristic different from a characteristic of thenon-defective product image and a known defect having a characteristicdesignated as the defective site, by inputting the workpiece image tothe machine learning network of which the parameter has been adjustedthrough the first learning process and the second learning process.

According to this configuration, not only the learning of the machinelearning network is performed by the defective product image, but alsothe learning of the machine learning network is performed using thenon-defective product image added with the noise, and thus, the machinelearning network having not only high detection capability for a knowndefect included in the defective product image used for learning butalso high detection capability for an unknown defect. As a result, alearning difficulty level is reduced as compared with a case where aninference process is performed in a non-defective product learning modeland a defective product learning model as in the related art, and thelabor at the time of learning can be reduced. In addition, a process ofcombining inference results is unnecessary at the time of the appearanceinspection, and thus, a takt time during the operation time becomesshort.

The processor according to another embodiment can execute the firstlearning process of causing an input image, obtained by adding a noiseto the non-defective product image, to be input to the machine learningnetwork and adjusting the parameter of the machine learning network suchthat an abnormality map indicating a position of the noise becomes afirst output image, and the second learning process of inputting thedefective product image, for which the designation of the defective siteby the user has been received, to be input to the machine learningnetwork and adjusting the parameter of the machine learning network suchthat an abnormality map indicating a position of the defective sitedesignated by the user becomes a second output image when performing asetting of the appearance inspection apparatus.

According to this configuration, it is possible to directly extract thedefective site as an abnormality in both non-defective product learningand defective product learning and output an abnormality map.

The processor according to still another embodiment can generate atarget abnormality map image, based on a difference in pixel valuesbetween corresponding sites of the non-defective product image addedwith the noise and a non-defective product image to which the noise isnot added, and adjust the parameter of the machine learning network suchthat the first output image coincides with the target abnormality mapimage during the first learning process, and thus, a learning effectusing the non-defective product image added with the noise is improved.

The processor according to still another embodiment randomly adds aplurality of the noises having a predetermined size or more to thenon-defective product image, and thus, detection performance of a finedefective site is improved while suppressing a portion, such as thevicinity of an edge of the workpiece, from being erroneously detected asthe defective site.

The processor according to still another embodiment adds the noise as acolor to the non-defective product image when the non-defective productimage is a color image, and thus, detection performance for anabnormality in color is improved.

The processor according to still another embodiment more increases theamount of the noise to be added to the non-defective product image asthe non-defective product image is larger, and thus, it is possible toautomatically add the amount of the noise suitable for the size of thenon-defective product image, and thus, it is possible to enhance thelearning effect while reducing the labor of the user.

The processor according to still another embodiment adds a plurality oftypes of the noises having different shapes to the single non-defectiveproduct image, and thus, it is possible to improve the detectionperformance for unknown defects having various shapes.

The processor according to still another embodiment executes an updateprocess of causing the machine learning network to learn an originaldata set to which a defective product image with annotation informationin which a defective site is designated by an annotation has been addedand updating the parameter of the machine learning network in a casewhere the workpiece image is an image obtained by capturing a defectiveproduct, but is not determined as the defective product, and thus, it ispossible to suppress detection omission of the defective site.

The processor according to still another embodiment executes, in a casewhere the workpiece image is an image obtained by capturing anon-defective product but is determined as a defective product as aresult of the inspection process, an update process of causing themachine learning network to learn an original data set to which theimage has been added as a non-defective product image and updating theparameter of the machine learning network, and thus, it is possible tosuppress the erroneous detection.

Since the machine learning network is trained with both the imageobtained by adding the noise to the non-defective product image and thedefective product image as described above, it is possible to detectboth the unknown defect having the characteristic different from acharacteristic of the non-defective product image and the known defectdesignated as the defective site. As a result, it is possible to stablyexhibit the high detection capability for the defective product imagehaving the unknown defect while shortening the takt time during theoperation time.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating a configuration of anappearance inspection apparatus according to an embodiment of theinvention;

FIG. 2 is a block diagram illustrating a hardware configuration of theappearance inspection apparatus;

FIG. 3 is a flowchart illustrating an example of a learning process of amachine learning network;

FIG. 4 is a diagram illustrating an input image, an output image, and atarget abnormality map image in the learning process of the machinelearning network;

FIG. 5 is a flowchart illustrating an example of a startup procedure ofthe appearance inspection apparatus;

FIG. 6 is a diagram illustrating a case where a workpiece image having aknown defect and an unknown defect is input to the machine learningnetwork; and

FIG. 7 is a flowchart illustrating an example of a procedure during theoperation time of the appearance inspection apparatus.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Hereinafter, an embodiment of the invention will be described in detailwith reference to the drawings. Note that the following description ofthe preferred embodiment is merely an example in essence, and is notintended to limit the invention, its application, or its use.

FIG. 1 is a schematic diagram illustrating a configuration of anappearance inspection apparatus 1 according to the embodiment of theinvention. The appearance inspection apparatus 1 is an apparatusconfigured to perform quality determination of a workpiece imageacquired by capturing a workpiece as an object to be inspected, such asvarious components and products, and can be used in a production sitesuch as a factory. Specifically, a machine learning network isconstructed inside the appearance inspection apparatus 1, and thismachine learning network is generated by learning a non-defectiveproduct image corresponding to a non-defective product and a defectiveproduct image corresponding to a defective product. The qualitydetermination of the workpiece image can be performed by the machinelearning network by inputting the workpiece image obtained by capturingthe workpiece as the object to be inspected to the generated machinelearning network.

All workpieces may be used as objects to be inspected, or only some ofthe workpieces may be used as objects to be inspected. In addition, oneworkpiece may include a plurality of objects to be inspected. Inaddition, a workpiece image may include a plurality of workpieces.

The appearance inspection apparatus 1 includes a control unit 2 servingas an apparatus main body, an imaging unit 3, a display apparatus(display section) 4, and a personal computer 5. The personal computer 5is not essential and can be omitted. Various types of information andimages can be displayed using the personal computer 5 instead of thedisplay apparatus 4, and a function of the personal computer 5 can beincorporated in the control unit 2 or the display apparatus 4.

In FIG. 1 , the control unit 2, the imaging unit 3, the displayapparatus 4, and the personal computer 5 are described as examples of aconfiguration example of the appearance inspection apparatus 1, but anyplurality of these may be combined and integrated. For example, thecontrol unit 2 and the imaging unit 3 can be integrated, or the controlunit 2 and the display apparatus 4 can be integrated. In addition, thecontrol unit 2 can be divided into a plurality of units and a partthereof may be incorporated into the imaging unit 3 or the displayapparatus 4, or the imaging unit 3 can be divided into a plurality ofunits and a part thereof can be incorporated into another unit.

[Configuration of Imaging Unit 3]

As illustrated in FIG. 2 , the imaging unit 3 includes a camera module(imaging section) 14 and an illumination module (illumination section)15, and is a unit that executes acquisition of a workpiece image. Thecamera module 14 includes an AF motor 141 that drives an imaging opticalsystem and an imaging board 142. The AF motor 141 is a portion thatautomatically executes focus adjustment by driving a lens of an imagingoptical system, and can perform the focus adjustment by a conventionallyknown technique such as contrast autofocus. The imaging board 142includes a CMOS sensor 143 as a light receiving element that receiveslight incident from the imaging optical system. The CMOS sensor 143 isan imaging sensor configured to be capable of acquiring a color image.Instead of the CMOS sensor 143, for example, a light receiving elementsuch as a CCD sensor can be used.

The illumination module 15 includes a light emitting diode (LED) 151 asa light emitter that illuminates an imaging region including aworkpiece, and an LED driver 152 that controls the LED 151. A lightemission timing, a light emission time, and a light emission amount ofthe LED 151 can be arbitrarily controlled by the LED driver 152. The LED151 may be integrated with the imaging unit 3, or may be provided as anexternal illumination unit separately from the imaging unit 3.

(Configuration of Display Apparatus 4)

The display apparatus 4 includes a display panel configured using, forexample, a liquid crystal panel, an organic EL panel, or the like. Aworkpiece image, a user interface image, and the like output from thecontrol unit 2 are displayed on the display apparatus 4. In addition,when the personal computer 5 includes a display panel, the display panelof the personal computer 5 can be used instead of the display apparatus4.

(Operation Equipment)

Examples of operation equipment configured for a user to operate theappearance inspection apparatus 1 include, but are not limited to, akeyboard 51, a mouse 52, and the like of the personal computer 5, andany equipment configured to be capable of receiving various operationsof the user may be used. For example, a pointing device such as a touchpanel 41 included in the display apparatus 4 is also included in theoperation equipment.

The control unit 2 can detect operations of the user on the keyboard 51and the mouse 52. In addition, the touch panel 41 is, for example, aconventionally known touch operation panel equipped with apressure-sensitive sensor, and a touch operation of the user can bedetected by the control unit 2. The same applies to a case where anotherpointing device is used.

(Configuration of Control Unit 2)

The control unit 2 includes a main board 13, a connector board 16, acommunication board 17, and a power supply board 18. The main board 13is provided with a processor 13 a. The processor 13 a controlsoperations of the connected boards and modules. For example, theprocessor 13 a outputs an illumination control signal for controllingon/off of the LED 151 to the LED driver 152 of the illumination module15. The LED driver 152 switches the on/off of the LED 151 and adjusts alighting time in response to the illumination control signal from theprocessor 13 a, and adjusts a light amount and the like of the LED 151.

In addition, the processor 13 a outputs an imaging control signal forcontrolling the CMOS sensor 143 to the imaging board 142 of the cameramodule 14. The CMOS sensor 143 starts capturing and performs thecapturing by adjusting an exposure time to an arbitrary time in responseto the imaging control signal from the processor 13 a. That is, theimaging unit 3 captures an image of the inside of a visual field rangeof the CMOS sensor 143 in response to the imaging control signal outputfrom the processor 13 a, and captures an image of a workpiece when theworkpiece is within the visual field range, but can also capture animage of an object other than the workpiece when the object is withinthe visual field range. For example, the appearance inspection apparatus1 can capture a non-defective product image corresponding to anon-defective product and a defective product image corresponding to adefective product by the imaging unit 3 as images for learning of amachine learning network. The image for learning is not necessarily animage captured by the imaging unit 3, and may be an image captured byanother camera or the like.

Meanwhile, the imaging unit 3 can capture an image of a workpiece duringthe operation time of the appearance inspection apparatus. In addition,the CMOS sensor 143 is configured to be capable of outputting a liveimage, that is, a currently captured image at a short frame rate at anytime.

When the capturing by the CMOS sensor 143 is finished, an image signaloutput from the imaging unit 3 is input to and processed by theprocessor 13 a of the main board 13, and stored in a memory 13 b of themain board 13. Details of a specific processing content by the processor13 a of the main board 13 will be described later. Note that aprocessing apparatus such as an FPGA or a DSP may be provided on themain board 13. The processor 13 a may be integrated with the processingapparatus such as the FPGA or the DSP.

The connector board 16 is a portion that receives power supply from theoutside via a power connector (not illustrated) provided in a powerinterface 161. The power supply board 18 is a portion that distributespower received by the connector board 16 to the respective boards,modules, and the like, and specifically distributes power to theillumination module 15, the camera module 14, the main board 13, and thecommunication board 17. The power supply board 18 includes an AF motordriver 181. The AF motor driver 181 supplies drive power to the AF motor141 of the camera module 14 to implement autofocus. The AF motor driver181 adjusts power to be supplied to the AF motor 141 in accordance withan AF control signal from the processor 13 a of the main board 13.

The communication board 17 is a portion that executes communicationbetween the main board 13, and the display apparatus 4 and the personalcomputer 5, communication between the main board 13 and external controlequipment (not illustrated), and the like. Examples of the externalcontrol equipment include a programmable logic controller and the like.The communication may be performed in a wired or wireless manner, andany communication form can be implemented by a conventionally knowncommunication module.

The control unit 2 is provided with a storage apparatus (storagesection) 19 configured using, for example, a solid state drive, a harddisk drive, or the like. The storage apparatus 19 stores a program file80, a setting file, and the like (software) for enabling each controland processing, which will be described later, to be executed by thehardware. The program file 80 and the setting file are stored in astorage medium 90, for example, an optical disk or the like, and theprogram file 80 and the setting file stored in the storage medium 90 canbe installed in the control unit 2. The program file 80 may bedownloaded from an external server using a communication line. Inaddition, the storage apparatus 19 can also store, for example, theabove-described image data, parameters for constructing a machinelearning network of the appearance inspection apparatus 1, and the like.

That is, the processor 13 a of the appearance inspection apparatus 1 isconfigured to read parameters and the like stored in the storageapparatus 19 to construct a machine learning network, and input aworkpiece image obtained by capturing a workpiece as an object to beinspected to the constructed machine learning network to perform qualitydetermination of the workpiece based on the input workpiece image. Theuse of the appearance inspection apparatus 1 enables execution of anappearance inspection method for performing quality determination of aworkpiece based on a workpiece image.

(Learning Process of Machine Learning Network)

Next, a learning process of a machine learning network performed at thetime of setting the appearance inspection apparatus 1 will be describedwith reference to a flowchart illustrated in FIG. 3 . The learningprocess of a machine learning network is to adjust parameters of themachine learning network by inputting a non-defective product imagecorresponding to a non-defective product and a defective product imagecorresponding to a defective product to the machine learning network forlearning.

In step SA1 after the start, an untrained machine learning network isprepared. In the untrained machine learning network, for example,initial values of parameters are randomly determined. Alternatively, amachine learning network trained to some extent may be prepared inadvance for appearance inspection.

In step SA2, a non-defective product image corresponding to anon-defective product is acquired. The non-defective product imageacquired here is a non-defective product image for learning illustratedin FIG. 4 , and may be a color image or a black-and-white image. Forexample, the non-defective product image can be acquired by capturingthe workpiece as the non-defective product by the camera module 14 ofthe imaging unit 3. Only one non-defective product image may beacquired, or a plurality of non-defective product images may be acquiredby capturing different non-defective products. The acquirednon-defective product image is stored in the storage apparatus 19, forexample.

In addition, in step SA3, a defective product image corresponding to adefective product is acquired. The defective product image acquired hereis a defective product image for learning illustrated in FIG. 4 , andmay be a color image or a black-and-white image. For example, thedefective product image can be acquired by capturing the workpiece asthe defective product by the camera module 14 of the imaging unit 3.Only one defective product image may be acquired, or a plurality ofdefective product images may be acquired by capturing differentdefective products. The acquired defective product image is stored inthe storage apparatus 19, for example. Steps SA1 to SA3 are notnecessarily executed in the above-described order.

In step SA4, a noise is added to the non-defective product image. FIG. 4illustrates a first input image input to the machine learning network asan image for learning, and this first input image is an image generatedby adding a noise to the non-defective product image for learning.Conventionally, as a technique of adding a noise to an image, atechnique of adding a single pixel noise according to a Gaussiandistribution is commonly used. In this technique, however, anon-defective site, such as the vicinity of an edge of a workpiece, islikely to be erroneously detected as a defective site. In the presentembodiment, a technique completely different from the conventional noiseadding technique is adopted. That is, the processor 13 a randomly adds aplurality of noises having a predetermined size or more to be largerthan a single pixel to the non-defective product image for learning,instead of the single pixel. A shape of the noise may be a circle, anellipse, a polygon such as a rectangle, or may be any shape. Inaddition, a plurality of types of noises having different shapes may beadded to the single non-defective product image for learning. At thistime, the processor 13 a more increases the amount of the noise to beadded to the non-defective product image for learning as thenon-defective product image for learning is larger.

In addition, conventionally, a gray noise is generally added when anoise is added to an image, but it is difficult to detect a colorabnormality in a case where the gray noise has been added. In thepresent embodiment, when the non-defective product image for learning isa color image, the processor 13 a adds a noise as a noise to thenon-defective product image for learning. The color noise is a chromaticcolor noise, and is noise of a color other than white, black, and gray(ashy color). In a case where a plurality of noises are added to thenon-defective product image for learning, a color may be changed foreach of the noises, or the same color may be used. In addition, when thenon-defective product image for learning is a black-and-white image, agray noise may be added. A site to which a noise has been added on thenon-defective product image for learning is an abnormal site.

After the noise is added, the process proceeds to step SA5. In step SA5,a first target abnormality map image (illustrated in FIG. 4 ) isgenerated based on a difference in pixel values between correspondingsites of the non-defective product image for learning (first input imageillustrated in FIG. 4 ) to which the noise has been added in step SA4and the non-defective product image for learning to which no noise isadded. Specifically, an average of absolute values of differencesbetween the added noise (abnormality) and a site corresponding to thenoise in the original non-defective product image for learning iscalculated, and a predetermined gain is applied to the calculatedaverage value. As a result, it is possible to obtain the first targetabnormality map image in which a pixel value of a portion other than thesite to which the noise has been added is zero. In this example, a largenumber of small circular noises are added, and thus, the correspondingto the noise is white and the other portion is black (whose pixel valueis zero) in the first target abnormality map image.

In addition, in step SA6, an annotation is executed on the defectiveproduct image acquired in step SA3. That is, the user designates thatthe defective product image acquired in step SA3 is an imagecorresponding to the defective product. A method for such designation isnot particularly limited, and examples thereof include a method ofadding a label indicating the defective product image. As a method ofadding a label, labels may be added to defective product images one byone, or for example, a plurality of defective product images may bestored in a specific folder, and a label may be collectively given tothe defective product images in the folder. The processor 13 a can storethe defective product image and the label, which is defect informationinput by the user, in the storage apparatus 19 in association with eachother.

In addition, the annotation also includes that the user designates adefective site in the defective product image. For example, theannotation may be an annotation of a region designation scheme in whichthe user encloses a defective site of a defective product imagedisplayed on the display apparatus 4 to designate the defective site, ormay be an annotation of a precise designation scheme in which adefective site is designated in an arbitrary shape by tracing thedefective site of a defective product image displayed on the displayapparatus 4. The user can select either the region designation scheme orthe precise designation scheme.

In the region designation scheme, the user operates the mouse 52 togenerate a frame enclosing the defective site of the defective productimage. For example, the user can designate the defective site on thedefective product image by generating a rectangular, circular, orfree-form frame having a size enclosing the defective site.

In the precise designation scheme, when the user moves a filling tool soas to trace a defective site 202 a, a portion other than the defectivesite is less likely to be included in a designated region, and thus, amore precise annotation can be made as compared with the regiondesignation scheme described above. In addition to the filling tool, adefective site may be designated by a magnet tool. The magnet tool canbe moved by operating the mouse 52, and moves so as to be automaticallyattracted to the defective site when being moved to the vicinity of thedefective site. When the defective site is moved into a frame connectinga plurality of the magnet tools, it is possible to accurately designatethe defective site while reducing burden on the user.

In addition, a defective site may be designated by the GrabCut tool.When a defective site and a periphery of the defective site are enclosedby the GrabCut tool, this region is designated, and an automaticextraction scheme of automatically extracting the defective site in thedesignated region is executed. In the automatic extraction scheme, onlythe defective site is designated, and a region in the periphery of thedefective site is not designated, and thus, precise designation of thedefective site is automatically performed as indicated by a white circleon the right. As a result, the burden on the user can be reduced.

In the case of the GrabCut tool, however, there is a case where precisedesignation of a defective site fails, and in this case, even a regionin a periphery of the defective site is included. In such a case, theuser finely designates foreground/background by performing strokecorrection or click correction after the execution of the automaticextraction scheme.

In addition, a defective site may be designated by AI-assisteddesignation. In the case of the AI-assisted designation, an outline of adefective site is roughly designated and extracted, and then, the insideof such an extracted site is designated by a fill tool or the like. As aresult, the defective site can be automatically extracted. It is alsopossible to perform fine correction after the automatic extraction ofthe defective site.

In step SA7, a second target abnormality map image (illustrated in FIG.4 ) is generated based on the defective product image on which theannotation has been executed in step SA6. In this example, the defectivesite is a linear flaw, and thus, in the second target abnormality mapimage, the defective site appearing linearly is white, and a portionother than the defective site is black (whose pixel value is zero).

In step SA8, parameters of the machine learning network are adjusted.Specifically, the processor 13 a inputs the non-defective product imagefor learning to which the noise has been added in step SA4, the firsttarget abnormality map image generated in step SA5, the defectiveproduct image on which the annotation has been executed in step SA6, andthe second target abnormality map image generated in step SA7 to themachine learning network. The non-defective product image for learningand the defective product image constitute a data set. The parameteradjustment of the machine learning network may be performed by the user,may be performed by a manufacturer who manufactures the appearanceinspection apparatus 1, or may be performed on a cloud.

As illustrated in FIG. 4 , when the non-defective product image forlearning to which the noise has been added is input to the machinelearning network, a first output image corresponding to thenon-defective product image for learning to which the noise has beenadded is output from the machine learning network. The first outputimage is an abnormality map indicating a position of the noise. Theprocessor 13 a adjusts the parameters of the machine learning networksuch that the first output image coincides with the first targetabnormality map image. That is, the processor 13 a executes a firstlearning process of causing the machine learning network to learn thenon-defective product image added with the noise and adjusting theparameters of the machine learning network such that a portioncorresponding to the noise is extracted.

In addition, when the defective product image on which the annotationhas been executed is input to the machine learning network, a secondoutput image corresponding to the defective product image is output fromthe machine learning network. The second output image is an abnormalitymap indicating a position of the defective site designated by the user.The processor 13 a adjusts the parameters of the machine learningnetwork such that the second output image coincides with the secondtarget abnormality map image. That is, the processor 13 a executes asecond learning process of causing the machine learning network to learnthe defective product image corresponding to the defective producthaving the defective site and adjusting the parameters of the machinelearning network such that the defective site designated in advance bythe user is extracted on the defective product image.

The first learning process can be performed a plurality of times using aplurality of non-defective product images for learning to which a noisehas been added and a plurality of first target abnormality map imagesrespectively corresponding thereto. In addition, the second learningprocess can be performed a plurality of times using a plurality ofdefective product images on which an annotation has been executed and aplurality of second target abnormality map images respectivelycorresponding thereto.

When the parameters are adjusted in step SA8, a trained machine learningnetwork is generated. Thereafter, in step SA9, information forconstructing the machine learning network, such as the parametersadjusted in step SA8, is stored in the storage apparatus 19 or the like.

(Startup Procedure of Appearance Inspection Apparatus 1)

Although the trained machine learning network can be generated andstored in the storage apparatus 19 at the time of setting the appearanceinspection apparatus 1 as described above, a startup process of theappearance inspection apparatus 1 may be executed as in a flowchartillustrated in FIG. 5 .

Steps SB1 to SB5 of the flowchart illustrated in FIG. 5 are the same assteps SA1 to SA5 of the flowchart illustrated in FIG. 3 . In addition,steps SB6 and SB7 of the flowchart illustrated in FIG. 5 are the same assteps SA8 and SA9 of the flowchart illustrated in FIG. 3 . In step SB6,the processor 13 a inputs a non-defective product image for learning towhich a noise has been added in step SB4 and a first target abnormalitymap image generated in step SB5 to a machine learning network at thetime of adjusting parameters of the machine learning network. Therefore,learning using a defective product image is not performed in the firststep SB6.

In step SB8 of the flowchart illustrated in FIG. 5 , a verificationprocess is executed. In this verification process, the detectioncapability of the machine learning network whose parameters have beenadjusted in step SA8 of the flowchart illustrated in FIG. 3 is verified.That is, a case is conceivable in which the machine learning networkwhose parameters have been adjusted in step SA8 has been trained bylearning, but the degree of the learning is low. When the operation ofthe machine learning network in a state in which the degree of learningis low is started, there is a possibility of causing detection omissionof a defective product image or erroneous detection in which anon-defective product image is detected as a defect. Therefore, beforethe operation of a trained machine learning network, the detectioncapability of the machine learning network is verified such that thedetection capability can be increased when the detection capability isinsufficient.

In the verification process, a workpiece image obtained by capturing adefective product is prepared. The workpiece image may be an imageacquired before the verification process, an image newly acquired forthe verification process, or an image acquired during the operation timeof the appearance inspection apparatus 1. The workpiece image is usedfor verification, and thus, can also be referred to as a test image. Asillustrated in FIG. 6 , the processor 13 a inputs the workpiece image tothe trained machine learning network. An output image (abnormality map)corresponding to the workpiece image is output from the machine learningnetwork.

An example illustrated in FIG. 6 illustrates a case where a workpiecehas a first defective site B1 and a second defective site B2. That is, alarge number of small circular noises are added in the non-defectiveproduct image added with the noise illustrated in FIG. 4 , but the firstdefective site B1 in the workpiece image illustrated in FIG. 6 has onecircular shape. Thus, the first defective site B1 has a characteristicdifferent from a characteristic of the non-defective product image addedwith the noise illustrated in FIG. 4 , and thus, corresponds to anunknown defect that has not been learned even in the trained machinelearning network. However, a non-defective product image added with anoise including a shape of the first defective site B1 is input at thetime of learning, and thus, the first defective site B1 can be detectedas illustrated as an output image in FIG. 6 even if the first defectivesite B1 is the unknown defect for the machine learning network.

In addition, the second defective site B2 is almost the same as thedefective site designated by the annotation in the second input imageillustrated in FIG. 4 , and thus, the second defective site B2corresponds to a known defect having a characteristic designated as adefective site. Since the second defective site B2 is the known defectfor the trained machine learning network, the second defective site B2can be detected by the machine learning network. That is, the processor13 a inputs a workpiece image to the machine learning network whoseparameters have been adjusted by the first learning process and thesecond learning process described above and executes the process ofdetecting both the unknown defect having the characteristic differentfrom a characteristic of the non-defective product image added with thenoise and the known defect having the characteristic designated as thedefective site. The verification process may be executed using oneworkpiece image, or may be executed by sequentially inputting aplurality of mutually different workpiece images to the machine learningnetwork.

The processor 13 a is configured to be capable of executing aninspection process of determining that a workpiece of the workpieceimage is a defective product when at least one of the unknown defect andthe known defect is detected as a result of the detection process, anddetermining that the workpiece of the workpiece image is a non-defectiveproduct when none of the unknown defect and the known defect aredetected. Note that it may be determined whether or not at least one ofthe unknown defect and the known defect is detected at the time ofsetting without determining the defective product and the non-defectiveproduct.

After step SB8, the process proceeds to step SB9. In step SB9, it isdetermined whether or not there is detection omission as a result of theverification process in step SB8. If there is detection omission in theworkpiece image input to the machine learning network, the processproceeds to step SB10. On the other hand, if there is no detectionomission in the workpiece image input to the machine learning network,it is determined as NO in step SB9, and the process proceeds to stepSB11.

Examples of a case where it is determined as YES in step SB9 can includea case where a workpiece image input to the machine learning network isan image obtained by capturing a defective product but is not determinedas a defective product. In this case, in step SB10, the user executes anannotation on the workpiece image corresponding to the defective productinput to the machine learning network in step SB8. The annotation can beexecuted similarly to step SA6 of the flowchart illustrated in FIG. 3 .Through step SB10, a defective product image with annotation informationcan be acquired. The defective product image with annotation informationcan be added to a data set and stored in the storage apparatus 19 or thelike.

In step SB12, a second target abnormality map image is generated basedon the defective product image with annotation information as in stepSA7 of the flowchart illustrated in FIG. 3 . Next, the process proceedsto step SB6, and the processor 13 a inputs the defective product imagewith annotation information and the second target abnormality map imagegenerated in step SB12 to the machine learning network. At this time,the machine learning network is trained with the original data set towhich the defective product image with annotation information has beenadded. Then, an abnormality map indicating a position of a defectivesite designated by the user is output from the machine learning network.The processor 13 a re-adjusts the parameters of the machine learningnetwork such that the abnormality map output from the machine learningnetwork coincides with the second target abnormality map image. That is,the processor 13 a executes an update process of training the machinelearning network with the defective product image with annotationinformation in which the defective site has been designated by theannotation and updating the parameters of the machine learning network.

In addition, in step SB11, it is determined whether or not there iserroneous detection. If there is erroneous detection in the workpieceimage input to the machine learning network, the process proceeds tostep SB13. On the other hand, if there is no erroneous detection in theworkpiece image input to the machine learning network, it is determinedas NO in step SB11, and the process proceeds to step SB14. In step SB14,a result of the verification process is output and presented to theuser.

Examples of a case where it is determined as YES in step SB11 caninclude a case where a workpiece image input to the machine learningnetwork is an image obtained by capturing a non-defective product but isdetermined as a defective product. In this case, in step SB13, an image(erroneously detected non-defective product image) determined as adefective product despite being a non-defective product image isacquired, and the process proceeds to step SB6. In step SB6 afterpassing through step SB13, the processor 13 a inputs the non-defectiveproduct image acquired in step SB13 to the machine learning network forlearning. At this time, the machine learning network is trained with theoriginal data set to which the non-defective product image has beenadded. As a result, the update process of updating the parameters of themachine learning network can be executed.

(Specific Method of Learning of Machine Learning Network)

Next, an example of a specific method of learning of a machine learningnetwork will be described. For example, learning of a machine learningnetwork can be performed by minimizing a loss function. Although thereare various definitions of the loss, the Mean Square Error (MSE) can beexemplified.

$\begin{matrix}{{Loss} = {\frac{1}{n}{\sum\limits_{x,y}\left( {T_{x,y} - O_{x,y}} \right)^{2}}}} & \left\lbrack {{Formula}1} \right\rbrack\end{matrix}$

Here, T is a target abnormality map, 0 is an output image (abnormalitymap), n is the number of pixels in which the image T is 0, and x and yare pixel positions. Note that a loss function such as the Binary CrossEntropy can also be used. The above is merely an example, and a learningmethod of a machine learning network is not limited to these methods.

(Operation Time of Appearance Inspection Apparatus 1)

Next, the operation time of the appearance inspection apparatus 1 willbe described based on a flowchart illustrated in FIG. 7 . In step SC1after the start, the processor 13 a reads parameters and the like storedin the storage apparatus 19 to prepare a trained machine learningnetwork. In step SC2, the imaging unit 3 captures an image of aworkpiece as an object to be inspected to acquire a workpiece image.Thereafter, the process proceeds to step SC3, and the workpiece imageacquired in step SC2 is input to the machine learning network preparedin step SC1.

Next, in step SC4, the machine learning network executes an inferenceprocess of the workpiece image input in step SC3. Thereafter, in stepSC5, the machine learning network outputs an abnormality map as a resultof the inference process. The abnormality map indicates the presence orabsence of an unknown defect having a characteristic different from anon-defective product image added with a noise, and the presence orabsence of a known defect having a characteristic designated as adefective site.

Thereafter, in step SC6, the quality of the workpiece is determinedbased on the abnormality map output in step SC5. That is, it isdetermined that the workpiece is a defective product when at least oneof the unknown defect having the characteristic different from acharacteristic of the non-defective product image added with the noiseand the known defect having the characteristic designated as thedefective site is detected, and it is determined that the workpiece is anon-defective product when none of the unknown defect and the knowndefect are detected. This quality determination is performed by theprocessor 13 a. A result of the quality determination of the workpiececan be output to, for example, the display apparatus 4 or the like to bepresented to the user, and can be stored in the storage apparatus 19.Note that steps SC2 to SC6 can be executed each time a workpiecechanges.

(Functions and Effects of Embodiment)

As described above, the learning of the machine learning network is notperformed only with the defective product image on which the annotationhas been executed, but the learning of the machine learning network canbe performed also using the non-defective product image added with thenoise. Therefore, it is possible to generate the machine learningnetwork having not only high detection capability for a known defectincluded in the defective product image used for learning but also highdetection capability for an unknown defect, and thus, a learningdifficulty level is reduced as compared with a case where the inferenceprocess is performed in both the non-defective product learning modeland the defective product learning model as in the related art, so thatthe labor at the time of learning can be reduced, and a process ofcombining inference results is unnecessary at the time of appearanceinspection, and thus, a takt time during the operation time isshortened.

The above-described embodiment is merely an example in all respects, andshould not be construed as limiting. Further, all modifications andchanges belonging to the equivalent range of the claims fall within thescope of the invention.

As described above, the invention can be used in the case of inspectingan appearance of a workpiece.

What is claimed is:
 1. An appearance inspection apparatus comprising: astorage section that stores a machine learning network; and a processorthat inputs a workpiece image obtained by capturing a workpiece, whichis an object to be inspected, to the machine learning network anddetermines quality of the workpiece based on the input workpiece image,wherein the processor is configured to be capable of executing: a firstlearning process of adding a noise to a non-defective product imagecorresponding to a non-defective product to cause the machine learningnetwork to learn the non-defective product image added with the noise,and adjusting a parameter of the machine learning network such that aportion corresponding to the noise is extracted; a second learningprocess of causing the machine learning network to learn a defectiveproduct image corresponding to a defective product having a defectivesite, and adjusting the parameter of the machine learning network suchthat the defective site designated in advance by a user on the defectiveproduct image is extracted; and a process of detecting both an unknowndefect having a characteristic different from a characteristic of thenon-defective product image and a known defect having a characteristicdesignated as the defective site, by inputting the workpiece image tothe machine learning network of which the parameter has been adjustedthrough the first learning process and the second learning process. 2.The appearance inspection apparatus according to claim 1, wherein theprocessor executes, when performing a setting, the first learningprocess of causing an input image, obtained by adding a noise to thenon-defective product image, to be input to the machine learning networkand adjusting the parameter of the machine learning network such that anabnormality map indicating a position of the noise becomes a firstoutput image, and the second learning process of causing the defectiveproduct image, for which the designation of the defective site by theuser has been received, to be input to the machine learning network andadjusting the parameter of the machine learning network such that anabnormality map indicating a position of the defective site designatedby the user becomes a second output image.
 3. The appearance inspectionapparatus according to claim 2, wherein during the first learningprocess, the processor generates a target abnormality map image, basedon a difference in pixel values between corresponding sites of thenon-defective product image added with the noise and a non-defectiveproduct image to which the noise is not added, and adjusts the parameterof the machine learning network such that the first output imagecoincides with the target abnormality map image.
 4. The appearanceinspection apparatus according to claim 1, wherein the processorrandomly adds a plurality of the noises having a predetermined size ormore to the non-defective product image.
 5. The appearance inspectionapparatus according to claim 1, wherein the processor adds the noise asa color to the non-defective product image when the non-defectiveproduct image is a color image.
 6. The appearance inspection apparatusaccording to claim 1, wherein the processor more increases an amount ofthe noise to be added to the non-defective product image as thenon-defective product image is larger.
 7. The appearance inspectionapparatus according to claim 1, wherein the processor adds a pluralityof types of the noise having different shapes to the singlenon-defective product image.
 8. The appearance inspection apparatusaccording to claim 1, wherein the processor executes an update processof causing the machine learning network to learn an original data set towhich a defective product image with annotation information in which adefective site is designated by an annotation has been added andupdating the parameter of the machine learning network, in a case wherethe workpiece image is an image obtained by capturing a defectiveproduct, but is not determined as the defective product.
 9. Theappearance inspection apparatus according to claim 1, wherein in a casewhere the workpiece image is an image obtained by capturing anon-defective product but is determined as a defective product as aresult of the inspection process, the processor executes an updateprocess of causing the machine learning network to learn an originaldata set to which the image has been added as a non-defective productimage and updating the parameter of the machine learning network.
 10. Anappearance inspection method of inputting a workpiece image obtained bycapturing a workpiece, which is an object to be inspected, to a machinelearning network and determining quality of the workpiece based on theinput workpiece image, the appearance inspection method comprising: afirst learning process of adding a noise to a non-defective productimage corresponding to a non-defective product to cause the machinelearning network to learn the non-defective product image added with thenoise, and adjusting a parameter of the machine learning network suchthat a portion corresponding to the noise is extracted; a secondlearning process of causing the machine learning network to learn adefective product image corresponding to a defective product having adefective site and adjusting the parameter of the machine learningnetwork such that the defective site designated in advance by a user onthe defective product image is extracted; and a process of detectingboth an unknown defect having a characteristic different from acharacteristic of the non-defective product image and a known defecthaving a characteristic designated as the defective site, by inputtingthe workpiece image to the machine learning network of which theparameter has been adjusted through the first learning process and thesecond learning process.